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ABSTRACT 

The program described here was a special short-term 
academic program at the University of Il}.inois for third-year 
students in the College of International Relations at Nihon 
University. Its objectives were to provide the students with 
effective training in^ English and an introduction to American history 
and culture which would be relevant to their university studies. The 
English classes in the 24-day program were held for four hours each 
day, vith the students divided into five levels according to their 
prof iciency test scores. The evaluation of the program focused on 
questions which addressed the following isi^ues: (1) needs and 
expectations, (2) degre f achievement and language learning 
motivation, (3) level ot language proficiency, (4) significant gains 
in English proficiency, (5) other knowledge and skills, (6) which of 
the participants* needs and; expectations were being met, (7) 
differences in participants* attitudes, (8) adequacy of the program 
stafff (9) appropriateness of time frame, and (10) benefits to the 
participating institutions. The final section of the report presents 
conclusions and recommendations, and appendices which include 
evaluation instruments in English and Japanese. (AMH) 
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Introduction 

This evaluation effort was undertaken as part of iry work as a graduate research 
assistant for the Program of Overseas University Collaboration, associated with the 
Office of Overseas Projects and Foreign Visitors (OPFV) at the University of Illinois 
ctt Urbana-Champaign . The objectives of the evaluation were to provide understanding 
regarding the program, to indicate areas of strength and weakness, and to aid in 
inproving future programs. Because of the nature of the program, the evaluation 
report may be of interest to other audiences, including the Intensive English Institute 
(lEI) at the University of Illinois, which was responsible for the ilish language 
conqponent, the participants, both staff and students, and other persons concerned with 
cross-cultural and international training programs. 

The e^'aluation offered me an opportunity to bring together a number of personal 
interests, including English as a second language, educational evaluation, and Asian 
studies. My escperience in these areas, supplemented by consultation and advice from 
experts in these fields, should provide a sound basis for cne value judgments made in 
this report. Most of the information on which these value judgments are based was 
provided by the participants in the program. I have made every effort to remain as 
objective and unbiased as possible in my in*''erpretation of this information. 

The report which follows attempts to describe the program ^ to explain the 
evaluation procedures, to interpret the information obtained, and to draw conclusions 
which address the questions posed for the evaluation. 
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Progreun Description 

Academic Interchange Agreement 

The 1981 Nihon University Overseas Training Program was the result of an academic 
interchange agreement between the University of Illinois at Urbana-Champaign and Nihon 
university at Mishima. This part: cular program waf. a special short-term academic pro- 
gram for third-year students in the College of International Relations at Nihon Univer- 
sity, and was ad-iinistered through the Program of Overseas University Collaboration, a 
program associated with the Office of Overseas Projects and Foreign Visitors at UIUC. 
The objectives of the program were to provide the students with effective training in 
English and an introduction to American history and culture which would be relevant to 
their university studies. 

Participants 

Most members of the junior class of the college (122 students) participated in 
the program. Other students in the college went to proyrams in China and the Philippines. 
They ranged in age from 20 to 22, and had an average of 9 years of formal English 
instruction. Eighty-four of the students were male, 38 female. The students were 
accompanied by four professors from their college and two escorts from Japan Travel* 
Bureau, who arranged transportation for the group. 

Dates 

The group arrived from Tokyo^ via Los Angeles and Chicago, on Ju-.e 28. Following 
the academic portion of their program, they left Urbana-Champaior. on July 23 for a two- 
day trip to the state capital Springfield, and to Macomb, Nauvoo, and Chicago, Illinois. 
On July 25, they left Chicago for short tours of Washington, D.C., New York, Los Angeles, 
San Francisco, and Honolulu, before returning to Tokyo on August 6. 

Activities 



The group spent twenty-four days on the Urbana-Champaign campus. Tlio firnt 
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days were spent in initial orientation activities, cultural research iessions under 
the direction of Harry C. Triandis, Director of the Program of Overseas University 
Collaboration, and English proficiency testing. English classes began on July 1. 
The classes were held in two sessions, 9:00 to 11:00 a.m. and 1:00 to 3:00 p.m. The 
students were divided into five levels, according to their proficiency test scores, 
and each level had classes focusing on 1) reading and discussion of readings, and 
2) situations and oral communication in situations. The content for these class 
sessions was aimed at providing the students with information which would be relevant 
to their interests and the activities of their program. During the academic portion 
of the program, other acti/ities were provided for the students. Lectures by Illinois 
faculty members in Law, Comparative Literature, History, Political Science, and Art 
and Design were held twice weekly. Other scheduled activities including dances, 
sports events, receptions, dinners, field trips to nearby parks and recreational 
areas, and a one-day trip to Chicago. On their last day on campus, the group had an 
English proficiency post- test. 

Staff 

The progrcun staff at the University of Illinois comprised ten English teachers, 
a staff secretary, two activities coordinators, and the directors of the Office of 
Overseas Projects and Foreign Visitors and Intensive English Institute. Oth^r 
Illinois staff had limited contact with the group as lecturers, researchers or 
residence hall managers. 

Facilities 

The Urbana-Champaign campus of the University of Illinois occupies over 700 
acres in central Illinois. It has over 180 major buildings, including excellent sports 
and recreation facilities and a hospital. The participants in this program were 
housed in the Illinois Street Residence Halls, an air-conditioned dormitory complex 
adjoining the central campus. Their meals were served in the cafeteria at the 
^'^ormltory, and their classes, were held in the Electrical Engineering building, vhich 





is aten-minute walk from the dormitory. The dormitory is situated on several major 
bus lines providing service to the entire twin-cities area of ^.^proximately 100,000 
residents. 
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Evaluation Procedures 

Evaluation Questions 

The evaluation focused on several questions which evolved through discussions 
with the directors of the Office of Overseas Projects and Foreign Visitors, the 
Intensive English Institute, and the Division of Measurement and Research of the 
Office of Instructional Resources. Assistance on the formulation of these questions 
was also provided by personnel in the Center for Instructional Research and Curriculum 
Evaluation, the Division of English as a Second Language, and the Office of International 
Programs and Studies. These questions were addressed: 

1. What are the perceived needs and expectations of the participants in this 
program? 

2. What relative degree of achievement motivation and language learning 
motivation exists in the group? 

3. What is the participants' relative level of English language proficiency? 

4. Were significant gains in English language proficiency achieved? 

5. What other knowledges and skills have the participants gained? 

6. Which of the participants* needs and expectations are being met? 

7. What 'iifferences exist in the participants* attitudes toward the different 
acti ities of the program? 

8. Is the program staff adequate in number and quality? 

9. Is the time frame appropriate? 
IG. What benefits accrue to the participating institutions? 

These questions were determined to be most responsive to the needs of the 
audiences for the evaluation, and had their support. In addition, the answers to 
these questions provided information directly related to the objectives of the evalua- 
tion effort. 

The answers to these questions may be interpreted from a number of different per- 
spectives, including those of the professional language educator, the cross-cultural 
trainer, the institutions involved, and the participants. It is important to note 
that these different perspectives cannot always be recognized by a single evaluator. 
For this reason, the reader is encouraged to examine the data and their interpretation 
^'^arefully, and recognize that alternative interpretations may be made. 
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Sources of Data 

In order to answer the evaluation questions, information from a number of sources 
was required. The primary source was the participants themselves. The staff of the 
English language component was another very important source, because of the amount 
of contact the teachers had with their students. A third source of information was 
the overall program staff, including the activities coordinators and the directors of 
the OPFV and lEI . 

Data Collection Instruments 



Several types of information were required, and the instruments and methods used 
to gather it were necessarily varied. The perceived needs and expectations of the 
participants were assessed through the use of a short Needs and Expectations Question- 
naire (NEQ) . Data on the participants' English language proficieiicy was obtained 
through the administration of the English Placement Test (EPT) which was made available 
by .the Division of English as a Second Language. Information regarding the participants" 
motivation was collected through the use of instruments adapted from those developed by 
Gardner and Lambert^ for use with bilingual students (MOTQ) . In order to obtain infor- 
mation relevant to the other evaluation questions, interviews were most appropriate. 
Two participants interview schedules and one staff interview schedule were prepared for 
this purpose. One final questionnaire, developed to assess the attitudes of partici- 
pants toward aspects of other training programs at the University of Illinois, was used 
to gather general information from the participants (POUCQ) . Copies of all of these 
instruments, with the exception bf the EPT, which is a secure examination, are included 
in the Appendix. 

Datv^ Collection Methods 

The Needs and Expectations Questionnaire (NEQ) was given to all of the student 
participants upon arrival in Urbana. This short questionnaire simply asked the 

^Gardner, Robert C. and Wallace E. Lambert, Attitudes and Motivation in Se cond Language 
^ Learning, Rowley, Massachusetts: Newbury House, 1972. 
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students to indicate their strongest and weakest areas in English, and the area 
which they felt would be most useful to them in their work. It also asked them to 
estimate the number of hours per week they expected to spend in various activities 
included in the program. The questionnaires were collected the next morning at the 
beginning of the first cultural research session. 

Tne English Placement Test (EPT) was given to all students op June 30. It is a 
three-part test, with a multiple-choice section on structure, a cloze passage (a prose 
passage with selected words deleted), and a dictation. The EPT has been standardized, 
and has been shown to be a valid and reliable test of English proficiency. The EPT 
was also given to all students on July 22, at the end of their academic program. 

The Motiv^^-ion Que->tionnaire (MOTQ) was given to all students on July 1, at the 
first meeting of English classes. This questionnaire was an adaptation of f-ve scales 
developed by Gardner and Lambert for use in their French-Ennlish bilingual projects. 
The adapted scales were translated and back-translated by native Japanese speakers, 
then duplicated in Japanese and given to the students. The sections of the MOTQ 
attempted to assess the motivation orientation and intensity of the students and theii 
desire to learn English. 

Three interview schedules were used In this evaluation. The first two were used 
in interviews with two ranaom samples of 25 students each. These interview schedules 
differ slightly, but have several questions in common. The first sample of students 
was interviewed the week of July 6-10, the second the week of July 13-17. The interviews 
were all held in Wardall Hall Lounge in the Illinois Street Residence Hall complex. The 
interviews averaged 15 minutes. The entire staff of the proqram was also interviewed, 
using a third interview schedule. These interviews were held in the offices of OPT^V, 
lEI, or by telephone, and averager' slightly longer, about 25 minutes. 

The last source of information was a questionnaire which had previously been 
developed for use in the Program of Overseas University Collabor-atiun , a traininq 
program for foreign university administrator^. It asked the respondents to rate various 
^aspects of their traininq program on an B-point scale. This questionnaire (Voiu-Q) was 
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translated^ duplicated, and given to all students after the last administration of 
the EPT, and was collected immediately. ^ # 



Orgiitnization and Analysis of Data 

Information from the NEQ was tabulated immediately and the results made, available 
to the entire program staff on July 1. These results are illustrated in Figures 1 
through 4. It may be seen that this group of students considered themselves to be 
strongest in the areas. of reading and writing and weakest in the areas of speaking anc 
listening. 'In addition'? they felt that the areas of speaking and listening would be 
of greater usefulness to them in their studies and careers. Figure 4 shows that the 
students* expectations of the amount of time they would spend in the \ rious activit"^ 
of the program were about r^'ght, with the exception of hours spent in English class, 
which actually was over twice what they expected. This information was passed jpn to 
the teachers and staff with the expectation that it would help them to deal more 
effectively with the students* perceived needs and to understemd more fully the 
expectations of the students regarding the progreUn, 
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Students' Perceived Usefulness of Areas of £;ngll8h 
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Figure 4 

Students* Expected Hours per Week in Program Activities 

Data from the initial and final administrations of the EPT and from the MOTQ 

were available for all of the students. The MOTQ scores were divided into subscores 

on Sex, Motivation Orientation (2 scores), Motivation Intensity, and Desire to Learn 

English. Thes6 scores were analyzed, using the Control Data Corporation Cyber 175 

coBqpiiter at the University of Illinois, and the basic distribution statistics and 

correlations among the variables were computed. The i^SPSS software package was used 
2 

in the analysis. 

The EPT total scores for this group of students ranged from 35 to 114, with pre- 
and post-test means of 65.78 and 70.51, respectively. An EPT score at this level 

Nie, N. H., C. H. Bull, J. G. Jenkins, K. Steinbrenner , and D. H. Bent. SPSS; 

Statistical Package for the Social Sciences. New York; McGraw-Hill, 1975. 
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would place a student in ESL 109, Remedial Grainmar, the lowest course offered in the 

Division of English as a Second Language for students admitted to the University. The 

relative level of Ent,Ush proficiency represented by a score at this level is more than 

two standard deviations below the mean score of students who have taken the EPT in the 

past, who had achieved a minimum of 480 on the TOEFL (Test of English as a Foreign 

Language), and had been admitted to the University. The EPT was used to determine 

whether they needed any further coursework in English and to provide information on 

which placement decisions could be made. 

The MO*^ subscores gave some indication of the type and level of achievement 

motivation and language learning motivation in the group of students, but the mean 

scores on the subscales were all within one standard deviation of the scales' midpoints^ 

and thus have no statistical significance. The Motivation Orientation subscores both 

indicated a slight integrative motivation, as opposed to instrumental motivation. 

Basically, integrative motivation indicates a deeper interest in the language as a 

reflection of the culture and people who use it. Instrumental motivation indicates 

interest in the language as a tool to be used to achieve other goals. The Motivation 

ft 

Intensity subscore and the Desire to Learn English siabscore were also very near the 
midpoints of '-he scales^ indicating a moderate level of achievement and language 
learning motivation. Because this questionnaire has not been used in this form with 
other groups, it is impossible to interpret what the scores represent. In future 
programs, however* they should provide a basis for comparison. 

Gains in mean scores on ^he subtests of the EPT were obtained by this group of 
students. The subtest and EPT total mean scores, expressed in raw score points and 
as percentages, are shown in .aule 1. The gains in mean scores on each subtest and 
on the total EPT test were tested for significance using the t-test for dependent 
samples. This Information is shown in Table 2. 
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Pre-test Post- test Gain 

Raw % Raw % Raw % 



Structure 


58.68 


58 


70 


62.10 


62.10 


3 


42 


3 


.40 


Cloze 


6.36 


21 


20 


6.99 


23-30 


.63 


2 


.10 


Dictation 


.74 


5 


30 


1.42 


10.10 


.68 


4 


.90 


Total 


65.78 


45 


60 


70.51 


49.00 


4 


73 


' 3 


30 



Table 1 

Student Gains on English Placement Test 



Haw Score Critical Degrees of 

Gain t freedom Sign if icance 



Structure 


3 


.42 


4 


68 


101 


P< 


.001 


Cloze 


.63 


2. 


25 


101 


P< 


.05 > .02 


Dictation 


.68 


4 


86 


101 


P< 


.001 


Total 


4 


.73 


6 


06 


101 


P < 


.001 



Table 2 

Significance of Students' EPT Gain Scores 



While significant gains in English language proficiency were obtained for the 
group during their short stay, a stronq relationship between motivation and these gains 
was not established. Pearson product-moment correlation coefficients illustrating the 
relationship are shown in Table 3. In addition to these low and even negative 
correlations, multiple regression ctnalysis revealed that the motivation of the 
students accounted for only about 3% of the total gain. 





Motivation 
Orientation 1 


Motivation 
Orientation 2 


Motivational 
Intensity 


Desire to 
Learn English 


Structure Gain 


-.15 


.04 


-.13 


-.06 


Cloze Gain 


.05 


i. 


.19 


.12 


Dictation Gain 


-.04 


.08 


.03 


.13 


Total Gain 


-.13 


.04 


-.05 


.00 



Table 3 

Pearson Product-Moment Correlacions Between EPT Gain and Motivatiot. Scale Scores 
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Intex views of the two samples of students yielded information which was compiled 



and categorized in order to help describe the program and respond to the evaluation 



(questions « 



The students in this progrcun were most often from the city, and their families 
could oe characterized as middle to upper middle class. More than half of those inter- 
viewed expressed interest in American and Japanese history / and about a third expressed 
interest in American 2uid Japanese art. About three-fourths of the group gave inter- 
national relations as their major subject area, with the remainder divided about 
equally between economics and Japanese literature. They expected the progreun to be 
most valuable to them in the areas of English (90%) , social relationships (60%) , and 
academic or professional areas (45%) . About one-fourth said they would benefit by 
learning a new game or sport while they were here. 

All of the students interviewed said they felt the program was valuable to them. 
When asked why, two-thirds of them said that their English would improve, one-third 
said the experience in America was beneficial, and about one-half said that contacts 
with Americans were valuable. When asked about the value of specific areas of the 
program, virtually all of the students mentioned the English classes, about one-half 
mentioned the social activities, and about one-third mentioned tours and trips as 
beneficial. Only about one- fifth mentioned the lectures as an important program 
component, and very few felt that the initial orientation and testing or subject area 
activities were very important. 

When questioned about the staff of the program, the students ranked the lEI 
teaching staff 90% excellent or good (70% excellent), the administrative staff 90% 
excellent or g'X>d (50% excellent), and coordinators 75% excellent or good (50% excellent). 
They rated the lectures 75% good or fair (20% excellent). 

Regarding th^ size of the staff, about two- thirds of the students said it should 
be larger. About 80% said the group should be smaller. The students interviewed 
were about evenly divided on whether the program should be longer. Of those who felt 
it should be longer, about half suggest two months, about one- third suggested a 3-roonth 




18 



: -16- 

Almost 90% of the students interviewed said that they had adjusted well to the 
conmtinity and culture of Urbana-Champaign. Of those who said they had adjusted well, 
the most common reasons given were that there were plenty of opportunities for conver- 
sation, that Americans were kind and polite, and that the university campus was large, 
open, and clean, of those who felt that they had not adjusted well, the most common 
reasc , given were that there were too many Japanese here and that they had cctfranunica- 
tion problems and difficulty in mciking friends. 

When asked to comment on strengths of the program, th3 students interviewed 
mentioned the good teaching staff and small classes most often. They also frequently 
mentioned the opportunities for conversation with Americans. The physical facilities 
such as the dormitory and recreation facilities were also mentioned. The students 
mentioned four weaknesses of the program with equal frequency — that there were too many 
Japanese, that their schedule was too full, that the lectures were not understandable, 
and that the food in the dormitory was unfamiliar and too sweet. Other comments of 
the students interviewed included a need for more communication with Americans on an 
informal basis, and more contact and information about'^ American family life and 
customs. 

Interviews with the staff of the program yielded information which was also 
compiled and categorized in such a way as to increase understanding of the program 
and to address the evaluation questions. 

The English teaching staff reported a great variety of work experience, most 
commonly ESL teaching in the U.S. and overseas, hut including ESL program administra- 
tion, elementary and secon^^ary teaching, and private tutoring. The s'^aff , on the 
average, had over 11 years' teaching experience, and over 3 years' overseas teaching 
experience • All hold Masters' degrees in TESL or linguistics, and several are doing 
Ph.D. work in these areas. 

The program coordination staff members had much less experience, averaging about 
2 years' contact -with foreign students or cross-cultural programs. The coordinators' 
^•xpeiience was in the area^ of recreational programming, living abroad, and ESL teaching. 
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The ESL teaching staff reported that progr^ planning and curriculum development 
activities began one week prior to the arrival of the students, on June 22. On that 
date, the teaching staff met, and the basic design of the program was outlined by the 
IRI director. Th^ staff divided itself according to personal preference for high or. 
low level classes in reading or oral communication areas. The five teachers in each 
area then met together for the remainder of the week to plan and develop the curriculum 
for that area. All of the teachers said that the time frame for these activities was 
adequate, given the qualifications and experience of the staff. 

The coordination staff reported beginning work on program development as early 
as April 30, but little was done until the middle of M^. They did not consider the 
time frame to have been adequate for the program planning activities which had to be 
carried out. 

All of the English teaching staff considered their salaries to be adequate for 
the work they did. Most of the teachers, however, would have preferred payment to 
have been in two equal payments. Due \:o the timing of the program, the teachers were 
paid in two unequal payments, the smaller one first, which created financial problems 
for some of them. The teachers were very positive about the quality of their colleagues, 
and rated both cooperation among the teaching staff and the secretarial support as 
excellent. 

The coordination staff was divided regarding the adequacy of the salary, but 
agreed that the timing of payment was suitable. It should be noted that the coordina- 
tion staff members were 50% time graduate assistants (approximately $1000 total) , 
while the ESL teaching staff were full-time Visiting Teaching Associates (approximately 
$2000 total).. The coordination staff was also divided as to its own qualifications, 
Thejt^ rated their cooperation as adequate, but it was much lower than that reported by 
the ESL teaching staff. 

The program staff generally felt that the size of the group of students was 
suitable. All those who did not agree felt that the size should be smaller. All felt 
.*-hat the program should be longer, and over three- fourths fe]t that a G-week program 
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would be about the right length. Regarding the physical facilities, the staff felt 
that the Electrical Engineering Building was not the best place" to have Ei glish classes 
during the sumner, because the building is not air-cpnditioned, most of the classrooms 
have immovable chairs, and access to video equipment is very difficult to arrange. 

The staff rated the motivation of the group of students as slightly betcer than 
average. All felt that the students* proficiency in English had iuprov^ during the 
program, sp^^^cif ically in the areas of listening and speaking. They also felt that the 
students had gained other knowledges and skills during the program, and mentioned social 
interaction skills and strategies, independence, and confidence in their ability to 
communicate in English. 

The staff members felt that the program was of benefit to them in a number of ways. 
First of all, they considered it to have been a valuable learning experience which was 
both enjoyable and increased their cultural sensitivity. They also felt that it had 
broadened their perspectives on cross-cultural education. ^ 

They felt that it benefited the students primarily in giving them a good cross- 
cultural experience and contact with native speakers of English. They also felt that 
the students had enjoyed themselves and that the program had increased their independence. 

The program staff felt that the University of Illinois had benefited from the 
program by having international students on campus, by increasing international contacts 
and exposure, and by increasing the possibility for more programs of this sort* 

They felt that Nihon University benefited by improving the quality and e/cperience 

of its students. Nihon also probably benefited, according to the program staff, by 

having such a relationship with the University of Illinois. The staff also felt that 

the programs increased the internitional contacts and exposure of Nihon University, and 

increased faculty-student cooperation in the College of International Relations. 

Obviously, this is a biased viewpoint « It indicates a need for additional sources on 

> 

this particular question. 

The program staff felt that the major strengths of the program were the TEI staff 
and curriculum. They also felt that the lEI administrative staff had done an excellent 




21 



The major weaknesses mentioned by the program staff were the lectures and lack 
of comnuni cation between the coordinators and the lEI staff. They also suggested 
increased .staff -student and student-American social contacts, a more careful definition 
of what the program should be, and a program of greater length. 

The interviews of students and staff helped to bring th# program more into focus. 
A great deal of important information was obtained which is directly related to the 
evaluation of the program and to suggestions for improving future programs of this kind. 

Information from the final Program of Overseas University Collaboration Questionnaire 
was analyzed and means computed for each of the questions. The information from this 
questionnaire is illustrated in Figure 5. 
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Figure 5 

Students' Satisfaction with Program Components 
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The results of this questionnaire indicate that in general, satisfaction with the 
program components was very high. The only area which was not satisfactory was the food 
which was served in the dormitory. 

Validity of Information 

It is particularly important that the validity of the measurement instruments and 
procedures be established vis-a-vis the objectives and content of the program, and 
assessed in relation to the evaluation questions posed for the evaluation. 

Some of the measures and procedures used in this evaluation were developed or 
adapted for this specific program, others are widely used for programs of this type. 

The NEQ (see Appendix) was developed for this program. Its validity depends 
entirely upon the willingeness of the respondents to answer honestly regarding their 
perceived strengths and weaknesses in English, the perceived usefulness of particular 
areas of English, and their expectations regarding the content areas of their program. 
It was administered before the program began, the respondents were assured that their 
answers would remain confidential, and they were informed that tlie information would 
be used to help make the program more responsive to their needs. The questionnaire 
was intended to gather information about the needs and expectations of the participants, 
and appears to have produced valid information in these areas. 

The EPT battery used in this evaluation was developed during the spring and 
summer terms, 1979, and was originally validated and normed with 195 foreign students 
entering the University of Illinois in fall, 1979. It has since been cross-validated 
and starciard score placement norms established with 270 foreign students entering Uie 
University of Illinois during the 1980 school year. The EPT battery was developed for 
the purpose of making esL placement decisions.^ Its use for assessing the language 
proficiency of the participants in this program seems warranted. It should be noted 
that no grades were assigned to the program participants. For this reason, it may be 
assumed that they were not under any great pressure to achieve gains in their EPT scores. 

O icjfflian, Lylc P. Personal Communication Regarding the EPT 
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The MOTQ (see Appendix) was adapted from scales develope^i by Gardner and Lambert in 
studies involving cross-cultural second language programs in Canada ^ Maine, Louisiana ^ 
and the Philippines. The scales were originally developed to assess motivation to learn 
French among native English speakers- Therefore, their appropriateness for the purposes 
of this evaluation may be questioned. The MOTQ was included in the evaluation procedures 
as the best available method for estimating the type and degree of motivation which 
existed in the group. However in measures of this sort, the "response set," or tendency 
to answer in a socially acceptable way, is often operant, so the responses of the 
students may be suspect. 

All of the interview schedules (see Appendix) were developed specifically for 
the participants in this program. The questions were intended to reveal information, 
relevant to the evaluation questions. Many of the same questions were asked of all 
the groups, in order to provide information on differences in perspectives of the 
groups. The respondents were assured of the confidentiality of their responses, and 
recognized that the information they supplied to be used to increase understanding 
of this program and to improve future programs of this type. 

The POUCQ (see Appendix) was intended to provide a general measure of the 
participants' satisfaction with specific areas of their program. It was administered 
following the final English testing session. Like the other attitude measures its 
accuracy depends on the participants' f orthrightness. The participants* names were not 
required on this questionnaire, and the program had come to an end. Tt may thus be 
assumed that the responses on the POUCQ reflect their true feelings about the program. 

Reliability of Information 

Since most of the instruments and procedures used in this evaluation were 
developed or adapted for this particular program, very little information on the 
•eliability of the measures is available. It is impossible at this point to determine 
the consistency which could be expected among similar observations using these instru- 
ments. In order to overcome this lack of information regardi. j the reliability of the 
n^^-lnstruments used, and also to improve the validity of the information obtained, a 

KJC 
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number of different instruments and procedures was used. Information c the reliability 
of the EPT is available. Estimated reliabilities for the test, based on the 1980 norm 
group (N=270) , are as follows: Structure, .969; Cloze, .776? Dictation, .948; EPT total. 



4 
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Conclusions Regarding Procedures 

The procedures used in this evaluation, which have been described in detail, appear 
to be adequate. However, there are some areas which might be improved. First, it would 
be helpful to have had other sources of information, particularly from Nihon University, 
regarding the program's benefits to the participating institutions. Second, the 
questionnaire on motivation does not appear to have been adequate for assessing the 
motivation of the participants. Perhaps the fact that the questionnaire was an 
adaptation is reflected in the absence of correlations with English proficiency test 
score gains. Third, the reliability of the instruments and procedures needs to be 
established. Generally, however, the procedures used seem to be responsive to the 
needs of the audiences ii^, providing information on which judgments may be based. 



4 
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Conclusions and Recommendations 
Conclusions Regarding Evaluation Questions 

1. The participants expected a program stressing English language 
training and social activities involving Americans. They expressed 
a need for practical communication practice in situations relevant 
to their career goals. 

2. Achievement motivation and second language learning motivation among 
qroup members appears to have been only slightly above average, and 
shows little relation to English proficiency increases. 

3. The relative entering level of English language proficiency of the 
group was lower than expected. 

4. Significant gains in English language proficiency, as measured by the 
EPT, were achieved by the group. 

5. Participants gained knowledge and skills in interpersonal communication, 
cultural sensitivity, independence, and confidence in their ability tc 
use English. 

6. The participants' needs in the areas of speaking and listening were 
adequately met by the lEI program in English. The opportunities for 
social interaction with Americans were not numerous enough. Their 
needs ^^nd expectations in other program areas appear to have been met. 

7. The participants were particularly impress d with the lEI program in 
English, but ^felt that the lectures were of little use. They also 
felt that mo)^ opportunities for informal interaction with Americans 
should have been provided. 

8. The program staff, especially the English teaching staff, was adequate 
in number and quality. 

9. The time frame of the program was adequate, but probably should be 
lengthened by about two 'weeks fof optimum results. 

10. The participating institutions may be expected to benefit from this 
program through increased international contacts and improvement of 
student and staff quality and experience. 



Re c omme pd a t i on s 

1. More detailed information regarding the content of the program should 
be provided to the- students before they arrive on campus. This should 
help to avoid discrepancies between expectations and actual program 
activities. 
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2. Since motivation as measured by the MOTQ does not appear to have 
\ been a «i?gnif icant factor in EPT gains, either another means of 

assessing motivation should be considered, or the entire question 
of motivation dismissed as relatively unimportant for the evaluation 
of this prograun. If a similar method ia used to assess motivation, 
it is necessary, to have "baareline" da^^fon Japanese students in 
^ general for com^jarison. 

3. Information regarding the level of English proficiency *of the group 
should be provided to the lEI prior to the students* arrival on campus. 
This will enable the staff to provide a curriculum which is more 
respbnsive to. their needs. 

4. The opportunities for informal social interaction between participants 
and Americans should be expanded. 

5. The high quality of the lEI staff should be maintained with high 
priority. 

6. The content and presentation of the lecture series should oe carefully 
revised -in order to mAke the lectures understandable to the students 
and more directly related to their career interests. This r'ivision 
should be based on mor^ information from Nihot) University, 

7. The coordinating staff should take care to include the lEI staff in 
informal and 'torthal meetings 'with the students, and should improve 
the exchcinge and dissemination of .information among all persons 
concerned with the program. In addition ; the coordinating staff 
should have a designated leader to avoid duplication of effort and 
establish specific ^eas of responsibility. Finally, the coordinating 
staff should make every effort to increase the cross-cultural contacts 
provided for the participants. 

8. The length of the program s^iould be increased to six weeks. 

9. Some arrangement should be made to provide Japemese-style food 
occasionally. 



Generally, this program appears tb have been a very successful one. The 
results obtained indicate that it should be continued with few modifications. 
The satisfaction of both stur nt nnd staff participants'^was very high, and it 
appears that all concerned were very happy with the experience it provided. 
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Office of Inftrnatfonal Programs and Studies 

MOGKAM OF OVERSEAS UNIVERSITY COLLABORATSON 
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(217) 333 ?9«0 Tele«; 206957 



NEEDS AND EXPECTATIONS QUESTIONNAIRE 



(check one) 



NAME: 



Japanese Literature 
International Relations 



DIRECTIONS: This questionnaire asks you to- indicate those areas of English 
which you feel are your strongest, your weakest, and which will 
be Brtost useful to you in J^our career. It also asks what 2unounts 
of time yop expect to spend in the different areas of the summer 
program. Ttiis information is for our use only, and will not be 
used for grades or be released to anyone else. Please return it 
to Dan Robertson tomorrow morning at the Cultural Adjustment 
Research Session. 



What are your two strongest areas in English? 
strongest » 2) 



(strongest = 1, next to 



Speaking 

Understanding Speech 

Reading * 

Writing 

2. What is your weakest area in English? (use a ^mark) 

Speaking 

Understanding Speech 

Reading 

Writing 

^.3. Which areas of English do you feel will be most useful to you in your work 
after college? (most useful «1, next most useful » 2^ etc.) 

Speaking 

Understanding Speech 
—,-1^ Reading 
Writing 

4. What amount of time do^you expect to spend in these activities during your 
program at Illi;iois? (use hours per week) 

hrs/wk Introduction to program and testing 

hrs/wk Attending lectures by Illinois faculty 

hrs/wk Studying English in class 

hrs/wk Optional social activities (such as, sports, da: jing. picnics) 

hrs/wk Optional subject area activities (university departments) 

hrs/wk Touring (bus trips, tours) 

/ 

THANK YOU FwK YOUR HELP! 
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Welcome to the 1981 Overseas Summer Training Program. We 
hope that this progr^^ will be interesting and valuable to you. In 
order to help us judge the effectiveness of the program, we would like 
tc get some information from you r«garding your feelings about the 
program and studying English. This is not a test. There are no "right" 
or wrong answers. Your teachers will not see your answers to the 
questions, ar \ they vill not be used to assign grades of any kind. Please 
an?v,r the questions as accurately as you can. Thank you 
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Directions: Here is & dtateiaent with four possible answers 
given. Read the statement and then reuik the answers 
froai "l" to ••4'' as tliey refer to you. Mark for the 
answer- that is most applicable to you, "a," the next 
most a|>plic2U>le, and so on. if you have sane reason 
that is not included in the answers, write it in the 
space after "e.* If you include a personal reason, 
your rankings will be from "1** to ••5." 

I am studying English because: 

1 think it will be useful in getting a good job 
someday. 

I think it will help me to understand American 
people 4nd their way of life. 
It will allow me to meet and coirinunicate with 
more and varied people, 

A knowledge of two languages will make me a 
better educated perscm. 
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Directions: Here are four reasons frequently given by 
students for studying English. Read each reason ' * 
carefully and indicate how well it describes your 
feeling about studying English. Put an "X" anywhere 
on the line to indicate how well the reason describes 
your feeling. 

I am studying English because: 

a. I think it will be useful in getting a good job 
someday. 



Nor MY FEELING DEFINIT2L¥ 
AT ALL [ MY FEELIMG 

One needs a good knowledge of at least one foreign 
language in order to gain social recognition » 

NOT MY FEELING DEFINITBI* 
AT ALL MY FEELIKQ 

I feel that no one is really educated until he is 
fluent in the English language. 



NOT MY FEELING DEFINITELY 
AT ALL MY FEELING 

d. I need it in order to finish my university work. 

NOT MY FEELING DEFINITELY 
AT ALL MY PE5LIN0 
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Dlrectioniis Here are four reasons frequently given by 
at^mts for studying English* Bead each reason 
carefully and indicate how well it describes your 
feeling about studying iSnglish. Put an "x" anywhere 
on. the line to indicate how well the reason describes 
your feeling. 

X am studying English because: 

a* It will help me to understand the American people and 
their way of life better. 



b* 



c» 



NOT MY FEELING 
AT ALL 



DEFINITELY 
MY FEELING 



It will help me gain good friends more easily among 
English-speaking people. 



NOT MY FEELING 
AT ALL 



DEFINITELY 
MY FEELING 



It should help me to 
Americans do* 

NOT MY FEELING 
AT ALL 



fjin to think and behave like the 



DEFINITELY 
. / FEELING 



d. It will allow me to meet and communicate with more and 
varied people. 



NOT MY FEELING 
AT ALL 



DEFINITELY 
MY FEELING 



Directions: Piease answer these questions by placing 
an ••X" in the space to the left of the statement 



which applies to you. 



1. 



3. 



5. 



Con^ared to others in my English classes, I think Is 

i*. do more studying than most of them. 

b. do less studying than most of them. 

study about as much as most of them. 
I think about the words and ideas that I learn about 
in my English classes: 

a. once in awhile. 

b. hardly ever. 

c. very frequent ly. 

If English was not taught in this school, I would 
probably: 

a. not bother learning English at all. 

try to find English lessons somewhere else. 

c. try to learn English in everyday situations 

(for example, read English books and news- 
papers, try to speak English when possible, 
and go to English language movies) . 

d. none of these (explain) 

On the average, i spend about the following amount of 

time doing home study in English: (include all 

English homework) 

a, four hours per week. 

b. one hour per week* 

c. seven hours per week. 

none of these (Give approxijnate number of 

hours per week: hours.) 

Considering how I study English, I would say that I: 



a. 

"b. 



After I 

a. 

^b. 

c. 



do just enough work to get along, 
will pass because of luck or intelligence 
because I do not work very hard* 
really try to learn English, 
none of these (explain) 
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finish my university work, I will probably: 
try to use my English as much as possible, 
make no attempt to remember my English, 
continue to improve my English (for example, 
daily practice, night school, etc*) 
none of these (explain) 



ERLC 
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Directions J Please answer these questions by placing 
an ••X* on the line or in^the space to the left 
of th| statement which applies to you. 

1. Place an "X** anywhere along the line to indicate how 
much you like English compared to all your other 
courses: 



DIGLISH TS^MY ENGLISH IS MY 
\, LEAST PREFERRED MOST PREFERRED 
COURSE COURSE 

2. Vfeen you have an assignment to do in English, do you: 

^ * it immediately when you start your homework 

become ccMi^letely bored — — 

c:;\^ put it off until all your other homework is 

finished 

d. none of these (explain) 



During English classes, I: 

have a tendency to daydream about other things. 

b. become cOT5>letely bored. 

c. have to force myself to keep listening to the 

teacher. 

become completely interested in the subject. 



4. If I had the opportunity and knew enough English, I would 
read English newspapers and magazines: 

a. as often as I could 

b . fairly regularly 
c. probably not very often. \ 



d. never. 



After I have been studying English for a short tj^e, I 
find tliat I: 

ja. have a tendency to think about other things* 



Jt>. am interested enough to get the assignment done. 
j:. beccane very interested in what I am studying. 
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6. If I had the opportunity to change the way English 
is taught in my university, I would: - - 
a. increase the amount of English training 

required. 

^b. keep the amount of English training as it is. 

c*. decrease the amount of English training 

requi red. 

7. I believe English should be: ' 
a. taught to all university students 

b. taught only to those students who wish 

to study it. 

c. omitted from the university curriculum. . 

8* I find studying English: 
a. very interesting. 

^b. no more interesting than most subjects. 

c. not interesting at all. 




i 



36 



R^l^f? OUQrseaS Samwfh Vrasr\\r\^ 
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* -36- 
PARTICIPJWW^ INTERVIEW SCHEDULE I 



1 • M^lWB^^^^^^ 

2* Sex/Age 

3. Years of foarmal English Self eval Subj eval 

4. Native language/country 

5. Cultural background city /country * 

father's profession 

interest in Japanese arts, history 

interest in American arts, history 

6. Subject area ' International Relations 

Japanese Literature 

Ecc^omics 

7. Which areas of your program here at Illinois do you feel are most important? 
Initial orientation 

English language program 

Lectures 

Social activities 

Subject area activities 

Touring 

8. Do you feel that the program should be }.onger? 

9. Have you adjusted w&ll to the coBtnunity and culture of Champaign-Urbana? 
Wiy or why not? 

10. Conments about the program. Strengths and weaknesses: 

11. ' Ifhat do you expect to gain from this program? 

,itoad«f«ic/Professional Econoflilc 
Social/Political Linguistic 
lUKsreav tonal 
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PAR^CIfAMTS INTERVIEW SCHEDULE 2 

1. Naae 

2. Sex/ Age 

3. Subject Area International Relations^ 

Japanese Literature 
Economics 



4. Do you thin^ your program is valuable and will be helpful to you in the 

future? Why or why not? 
5* Which areas of the program do you feel are most beneficial?^ 

Initial orientation 

^English language progiram 

^Lectures 

^Social activities 

^Subject area activities 

^Touring 



6* Do you feel that the program should be longer? 
7* Please rate the staff of this program/s components: 
Administrative staff of Overseas Projects 

Excellent ^^^^ 

Intensive English Institute Staff 

Excellent^ ^oox 

Lecturers 

Excellent P oor 

Social/Subject Area/Touring Coordinators 
Excellent Poor 
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8- Should there be a larger or smaller group of you? 

larger or smaller staff? 

9, Have you adjusted well to the community/culture of Champalgn/Urbana 
Why or why not? 

10. Comments about the program. Strengths and weal^nesses. 

11, What do you expect to gain from this program 

Academic/Professional 

Social/Political 

Economic 
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Linguistic 



Recreational 
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staff Interview Schedule 

1. Ua m 

2. Sea^/Age_ 

3. Foeitio n# 



4* B3iperience: What kind of experience have you had for this job? 



How long have you wrked in this area? 



Is your experience closely related to the job you have now? 



5. Opinions regarding the program: 

Curriculum development/program planning activities: 

When did activities begin? 

Was the time frame adequate? 

Who participated in these activities? 

Administrati^|/coordination of program: 

Was your salary adequate? 

Was the timing of payment suitable? 

Do you feel that the program staff was qualified? 

Were staff support services adequate? 

Rate the cooperation among. staff members. 

Logistics: 

Would you prefer a larger or smaller group? 

Would you prefer a larger or smaller staff? 

Woi^d you prefer a longer or shorter program? 

Were the physical facilities adequate for the program? 

Students: 

Were the students well motivated? 

Do you think their English proficiency improved? 

Which areas inqproved most? 

Did they gain any other knowledges and skills? 
Which ones? 

Do you think they adjusted well to the culture? 

Benefits of the program: 

To you? How? 

To the students? How? 

To the U of I? How? 

To Nihon University? How? 

6. Comments regarding strengths and weaknesses of the program. 
?• Suggestions for improving future programs of this kind. 
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PROGRAM (Xf OVERSEAS JNIVERSITY COLLABORATION 
PARTICIPANTS QUESTIONNAIRE 

DIRECTIONS J Circle the nuiiA>er of the answer *rtiich i« applicable to your experience 
at the University of Illinpis. 
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1. 


«!• Intensive English Institute 
prograa in English wass 


1 


2 


3 


4 


5 


6 


7 


8 


2. 


The content of my training 
progrm wasi 


1 


2 


3 


4 


5 


6 


7 


8 


3. 


The p«ople «fho worked with me 
in ay training program were: 


1 


2 


3 


4 


5 




7 


8 


4. 


The POOC personnel werei 


1 


2 


3 


4 


5 


6 


7 


8 


5. 


The university facilities I saw 
weret 


1 


2 


3 


4 


5 


6 


7 


8 


6. 


Hm housing facilities in 
Chaapaign-Urbana were: 


1 


2 


3 


4 


5 


6 


7 


8 


7. 


The food wa«t 


1 


2 


3 


4 


5 


6 


7 


8 


6. 


The entertainsMmt and outside 
activities were I 


.1 


2 


3 


4 


5 


6 


7 


8 


f. 


fh« trips outside Chaaqpaign- 
Urtteaa wvret 


1 


2 


3 


4 


5 


6 


7 


8 


10. 


TIM qpipKrtUBities for informal 
oMtaett.with A»erieans iferet 


1 


2 


3 


4 


5 


6 


7 


8 
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